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Abstract 

Air  Force  Smart  Operations  for  the  21st  Century  (AFS021)  is  not  currently  being 
used  as  effectively  as  it  could  be  across  the  Air  Force.  Instead  of  trying  to  simply  save 
money  here  or  cut  man-hours  there,  AFS02 1  tools  should  be  used  to  help  the  Air  Force 
“fly,  fight  and  win”  better. 

The  Air  Force  Institute  of  Technology  (AFIT)  has  developed  a  methodology  to 
identify  specific  target  areas  where  continuous  process  improvement,  i.e.,  AFS021,  can 
be  applied  to  improve  the  bottom  line  of  an  organization.  The  first  step  of  this  process  is 
to  solicit  the  key  performance  indicators  (KPIs)  that  best  reflect  the  organization’s 
mission.  The  second  step  is  to  use  and/or  develop  metrics  based  on  those  KPIs  to 
measure  the  organization’s  mission  performance  today.  The  third  step  is  to  capture  the 
trends  of  those  KPIs  over  time  to  see  if  the  organization  is  getting  better  or  worse.  The 
final  step  is  to  identify  the  largest  perfonnance  capability  gaps  in  order  to  detennine 
where  AFS02 1  resources  should  be  applied  to  get  “the  most  bang  for  the  buck”. 

The  result  of  this  process  should  give  the  decision  maker  the  ability  to  improve 
the  bottom  line  of  an  organization  by  improving  its  weakest  areas.  Air  Combat 
Command  is  used  as  a  case  study  for  the  application  of  this  methodology. 
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METRIC  DEVELOPMENT  FOR  CONTINUOUS  PROCESS  IMPROVEMENT 


I.  Introduction 

Background 

The  use  of  Continuous  Process  Improvement  (CPI)  within  organizations  has 
become  very  popular  in  recent  years.  Since  early  pioneers  invented  machines  that 
enabled  and  improved  mass  production,  others  have  taken  additional  steps  forward,  such 
as  creatively  applying  statistical  methods  to  analyze  processes  and  improve  quality  by 
reducing  variation  within  processes.  Today,  some  companies  focus  their  CPI  efforts  on 
maximizing  the  quality  of  their  products  and  services  in  order  to  improve  customer 
satisfaction,  while  others  use  CPI  as  a  mechanism  for  driving  down  costs.  Still  others 
aim  to  change  the  culture  within  their  organization  to  one  that  rewards  its  members  for 
sharing  ideas  that  continually  improve  their  internal  business  processes  and  policies. 

In  general,  the  purpose  of  CPI  is  to  reduce  costs,  eliminate  waste,  increase 
efficiency,  improve  product  quality,  and  maximize  value  to  the  customer.  This  is  true  for 
profit  and  non-profit  organizations  alike.  For  the  Air  Force,  the  desired  effects  of  its  CPI 
initiative,  Air  Force  Smart  Operations  for  the  21st  Century  (AFS021),  are  to  increase 
productivity  of  its  personnel,  increase  critical  asset  availability,  improve  response  time, 
maintain  safe  and  reliable  operations,  and  improve  energy  efficiency.  Unfortunately, 
these  effects  have  not  yet  been  fully  realized  since  the  advent  of  AFS021. 
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Problem  Statement 


AFS02 1  is  not  currently  being  used  as  effectively  as  it  could  be  across  the  Air 
Force.  The  Air  Force  website.  Air  Force  Times,  and  other  media  sources  tell  about 
various  success  stories  where  processes  are  streamlined,  hundreds  of  man-hours  are  cut, 
and  thousands  of  American  tax  dollars  are  saved.  These  are  obviously  very  good  things, 
but  the  reader  needs  to  question  the  effect  that  these  scattered  (and  somewhat  random) 
improvements  have  on  the  core  mission.  In  other  words,  the  question  is:  how  can 
AFS021  be  used  to  get  away  from  creating  appealing  headlines  to  truly  making  a 
difference  for  the  Air  Force? 

Research  Objective 

The  purpose  of  this  research  is  to  find  a  way  to  use  CPI  tools  to  go  from  simply 
saving  money  here  or  cutting  man-hours  there  to  solving  the  bigger  issue  of  improving 
the  Air  Force’s  bottom  line.  For  a  business,  the  bottom  line  is  profit.  Therefore,  to 
improve  their  bottom  line  is  to  increase  profit.  However,  since  the  Air  Force  is  a  non¬ 
profit  government  entity,  it  can  be  argued  that  its  bottom  line  is  mission  performance.  In 
other  words,  the  Air  Force  need  to  use  CPI  tools  to  “fly,  fight  and  win”  better. 

Methodology 

This  research  provides  a  methodology  to  identify  specific  target  areas  where 
continuous  process  improvement,  i.e.,  AFS021,  should  be  applied  to  improve  the  bottom 
line  of  an  organization.  The  first  step  in  this  process  is  to  engage  the  decision  maker  to 
solicit  the  key  performance  indicators  (KPIs)  that  best  reflect  the  organization’s  mission. 
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The  goal  for  this  step  is  to  make  a  list  of  the  answers  to  the  “why  are  we  here”  question. 
The  second  step  is  to  use  and/or  develop  appropriate  metrics  based  on  those  KPIs  to 
measure  how  well  the  organization  is  doing  its  mission  today.  The  third  step  is  to  capture 
the  trends  of  those  KPIs  over  time  to  see  if  the  organization  is  getting  better  or  worse. 

The  fourth  and  final  step  is  to  identify  the  largest  performance  capability  gaps  in  order  to 
detennine  where  AFS02 1  resources  should  be  applied  to  get  “the  most  bang  for  the 
buck”.  The  result  of  this  process  should  give  the  decision  maker  a  clear  snapshot  of 
his/her  organization’s  current  ability  to  perform  its  mission.  This,  in  turn,  gives  him/her 
the  ability  to  improve  the  bottom  line  by  targeting  the  weakest  areas. 

Scope 

The  methodology  presented  in  this  paper  is  applicable  to  any  organization,  which, 
by  definition,  is  any  entity  that  pursues  collective  goals,  controls  its  own  perfonnance, 
and  has  a  boundary  separating  it  from  its  environment.  This  includes  profit  and  non¬ 
profit  organizations.  The  Air  Force’s  Air  Combat  Command  (ACC)  will  be  examined  as 
a  case  study  for  a  specific  application  of  the  proposed  methodology. 

Paper  Organization 

This  research  presents  a  framework  for  applying  CPI  in  order  to  improve  the 
bottom  line  of  an  organization.  In  Chapter  2,  the  research  begins  by  examining 
foundational  concepts  including  various  CPI  methodologies,  principles  of  metric 
development,  basic  tenets  of  regression  analysis,  and  background  infonnation  about 
ACC.  In  Chapter  3,  the  previous  concepts  are  combined  to  establish  a  methodology  for 
effectively  employing  CPI  in  an  organization.  Chapter  4  details  the  analysis  perfonned 
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and  results  achieved  through  application  of  the  methodology  to  ACC.  Finally, 
recommendations  for  further  study  and  future  implementation  are  provided  in  Chapter  5. 
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II.  Literature  Review 


Introduction 

The  purpose  of  this  literature  review  is  to  provide  a  foundation  for  the  discussion 
about  metric  development  for  Continuous  Process  Improvement  (CPI).  It  briefly 
introduces  some  common  CPI  methodologies,  culminating  with  a  description  of 
AFS021,  which  is  the  CPI  methodology  used  by  the  Air  Force.  It  also  describes  the  key 
tenets  of  metric  development  and  provides  a  description  of  the  Air  Combat  Command 
(ACC)  organization  which  will  be  used  as  the  case  study  for  the  methodology  in  this 
thesis. 

Continuous  Process  Improvement  (CPI) 

CPI  Defined 

Continuous  Process  Improvement  (CPI)  is  problem  solving.  It  refers  to  the 
integrated  system  of  improvement  that  organizations  use  to  analyze  and  improve  their 
internal  processes  on  a  recurring  basis  by  focusing  on  doing  the  right  things,  right  (Liker, 
2004).  This  concept  stems  from  the  Japanese  term  kaizen,  which  is  a  philosophy  that 
emphasizes  continuous  improvement  throughout  all  aspects  of  life;  kai  means  “change” 
and  zen  means  “good”.  As  a  way  of  thinking,  CPI  is  relevant  to  any  process,  regardless 
of  complexity  or  relative  importance  (Womack,  2003). 

The  Purpose  of  CPI 

The  goal  of  CPI  is  inherent  in  its  name;  to  continuously  improve  products  and 
services.  Knowledge  about  what  the  customer  wants  is  essential  to  achieving  this  goal 
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because  the  customer  determines  what  is  of  value  (Womack,  2003).  In  the  private  sector, 
the  goal  of  CPI  is  typically  to  reduce  costs  related  to  internal  processes  and  people  in 
order  to  increase  profits  and  provide  higher  quality  products  to  customers.  In  the 
military,  the  goal  is  to  achieve  lower  cost,  shorter  lead  times,  and  higher  quality  in  order 
to  deliver  better  products  more  quickly  to  the  warfighter  (AF  Journal  of  Logistics,  2008). 

CPI  Methodologies 

Some  of  the  CPI  programs  that  are  being  used  throughout  corporate  America 
include  Lean,  Six  Sigma,  Quality,  Business  Process  Reengineering  (BPR),  and  Theory  of 
Constraints  (TOC).  Within  the  DoD,  CPI  programs  have  service-specific  titles,  such  as 
the  Navy's  AIRSpeed,  the  Air  Force's  Smart  Operations  for  the  21st  Century  (AFS021), 
and  the  Anny's  Lean  Six  Sigma  (LSS)  program. 

Lean 

The  fundamental  premise  of  “lean”  is  the  elimination  of  muda,  or  waste,  in  the 
workplace.  Waste  can  be  defined  as  “any  activity  that  consumes  resources  but  creates  no 
value”  (Womack,  2003).  According  to  the  best-selling  book  The  Toyota  Way,  there  are 
eight  types  of  waste  in  the  workplace:  overproduction,  waiting,  unnecessary  transport  and 
conveyance,  over  processing  (or  incorrect  processing),  excess  inventory,  unnecessary 
movement,  defects,  and  unused  employee  creativity  (Liker,  2004).  The  purpose  of  being 
lean  is  to  be  able  to  identify  and  remove  these  types  of  waste  to  “do  more  and  more  with 
less  and  less  -  less  human  effort,  less  equipment,  less  time,  and  less  space  -  while 
coming  closer  and  closer  to  providing  customers  with  exactly  what  they  want”  (Womack, 
2003).  Lean  production  is  based  largely  on  the  Toyota  Production  System,  Toyota’s 
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unique  approach  to  manufacturing  that  allows  them  to  use  “fewer  man-hours,  less 
inventory  (to  produce)  the  highest  quality  cars  with  the  fewest  defects  of  any  competing 
manufacturer”  (Liker,  2004).  Toyota  pioneered  this  system  after  World  War  II  during  a 
time  when  American  companies  Ford  and  GM  were  using  mass  production  and  large 
equipment  to  produce  as  many  parts  as  possible  as  cheaply  as  possible.  Toyota,  however, 
had  a  much  smaller  customer  base,  and  had  to  produce  a  variety  of  vehicles  in  small 
quantities  using  the  same  assembly  line.  They  realized  that  keeping  lead  times  short  and 
focusing  on  flexible  production  lines  actually  resulted  in  higher  quality,  better 
productivity,  and  better  utilization  of  equipment  and  space  (Liker,  2004).  They  were,  in 
effect,  doing  more  with  less. 

The  fundamental  tenets  of  lean  production  are  value,  the  value  stream,  flow,  pull, 
and  perfection.  Value  is  defined  by  the  customer  as  the  good  or  service  that  is  being 
produced.  The  value  stream  refers  to  the  series  of  sequential  steps  that  the  value  takes 
through  a  process  to  reach  its  finished  state  when  it  is  ready  to  be  given  to  the  customer. 
Figure  1  shows  a  value  stream  map  for  a  company’s  steel  assembly  fabrication  process. 

In  this  example,  the  value  is  an  individual  part  made  out  of  steel  provided  by  “Michigan 
Steel  Co”  (the  supplier)  for  distribution  to  “State  Street  Assembly”  (the  customer).  Flow 
refers  to  the  ability  of  a  value  to  move  continuously  from  step  to  step  in  its  production 
process  without  interruption  or  wasted  motion;  it  is  the  contrary  method  to  a  batch-and- 
queue  process.  Pull,  in  its  simplest  terms,  means  that  an  item  should  not  be  produced 
until  it  is  requested  by  its  customer.  This  is  in  contrast  to  a  “push”  system,  which 
attempts  to  predict  customer  demand  and  cover  that  demand  with  inventory.  Pull  systems 
provide  the  right  amount  at  the  right  time;  no  more,  no  less,  not  early,  not  late.  Finally, 
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perfection  is  just  that  -  repeating  the  leaning  process  until  there  is  complete  elimination 
of  muda  (Womack,  2003).  Incorporating  these  tenets  into  a  single  effort  with  a  focus  on 
eliminating  wasted  time  and  resources  at  each  step  results  in  a  fast,  flexible  process  that 
gives  customers  what  they  want  when  they  want  it,  at  the  highest  quality  and  affordable 
cost  (Liker,  2004). 


Note:  C/T  =  cycle  time;  C/O  =  change-over  time;  EPE  =  every  part  every 
Source:  Womack  (2003) 


Figure  1.  Example  of  a  Value  Stream  Map  (Womack,  2003) 


Six  Sigma 

While  Lean  focuses  on  making  a  process  more  efficient,  Six  Sigma  is  a 
methodology  used  to  improve  product  quality.  Six  Sigma  is  defined  as  a  comprehensive 
and  flexible  system  for  achieving,  sustaining  and  maximizing  business  success.  Six 
Sigma  is  uniquely  driven  by  close  understanding  of  customer  needs,  disciplined  use  of 


facts,  data,  and  statistical  analysis,  and  diligent  attention  to  managing,  improving,  and 
reinventing  business  processes  (Pande,  2000).  Six  Sigma  was  embraced  by  the  Motorola 
Corporation  as  a  method  of  improving  customer  satisfaction  by  increasing  product 
quality.  “Sigma  (a)”  is  a  term  in  statistics  that  refers  to  the  amount  of  variation  of  data 
points  from  the  mean  in  a  data  set.  Within  the  context  of  a  normal  distribution,  this 
means  that  99.99998%  of  all  data  points  are  within  six  standard  deviations  (six  sigma) 
from  the  mean  (see  Figure  2).  In  business  terms,  this  can  be  quantified  as  operating  with 
only  3.4  defects  per  million  opportunities,  where  a  “defect”  is  defined  as  any  instance  or 
event  in  which  the  product  or  process  fails  to  meet  a  customer  requirement.  A  company 
that  is  able  to  fine-tune  its  products  and  processes  to  this  level  will  be  near-perfection  in 
meeting  customer  requirements  (Pande,  2000). 


Figure  2.  "Six  Sigma"  deviations  from  the  mean  (Pande,  2000) 


The  Six  Sigma  model  used  to  guide  process  improvement  is  called  the  “DMAIC” 
model,  which  is  a  five-step  process  used  to  Define,  Measure,  Analyze,  Improve,  and 
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Control  product  and  process  defects  (Pande,  2000).  Figure  3  depicts  the  “DMAIC” 
model  below: 


Figure  3.  "DMAIC"  model  (Pande,  2000) 

The  DMAIC  model  consists  of  the  following  five  steps: 

•  Define  the  process  by  identifying  the  customers,  identifying  what  is  important  to 
the  customers,  and  identifying  existing  output  conditions. 

•  Measure  the  process  by  using  metrics  to  collect  data. 

•  Analyze  the  data  results  to  identify  the  most  important  causes  of  the  problems. 

•  Improve  the  process  by  developing  and  implementing  solutions. 

•  Control  the  process  means  that  once  the  process  is  within  performance  standards, 
it  is  monitored. 


Lean  Six  Sigma 

Lean  Six  Sigma,  as  the  name  implies,  combines  Lean  and  Six  Sigma  to  achieve 
greater  process  improvement  gains.  The  purpose  of  Lean  is  to  minimize  waste  (increase 
efficiency)  and  that  of  Six  Sigma  is  to  reduce  variation  (increase  effectiveness).  The 
result  of  the  combination  is  the  customer  will  receive  a  defect- free  product  faster.  In 
general,  Lean  techniques  will  result  in  more  immediate  gains.  Improvements  from  Six 
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Sigma  application  will  take  longer.  Using  the  methods  together  will  maximize 
productivity  and  ensure  customers  are  getting  what  they  need,  when  they  need  it  (George, 
2002). 

Lean  and  Six  Sigma  principles  can  be  applied  to  logistics  and  supply  chain 
operations.  In  their  book  Lean  Six  Sigma  Logistics:  Strategic  Development  to 
Operational  Success,  Thomas  Goldsby  and  Robert  Martichenko  developed  a  useful  tool 
they  refer  to  as  the  “bridge”  model  (see  Figure  4). 


Figure  4.  Lean  Six  Sigma  Logistics  "Bridge"  Model  (Goldsby,  2005) 

The  guiding  principles  of  this  model  for  solving  logistics  challenges  are  logistics 
flow,  logistics  capability,  and  logistics  discipline,  which  form  a  “bridge”  between  a 
company  and  its  customers.  Logistics  flow  refers  to  the  movement  of  assets,  information, 
and  financial  data  across  the  bridge.  Logistics  capability  encompasses  predictability, 
stability,  and  visibility  of  the  company’s  processes.  Logistics  discipline  refers  to 
collaboration,  systems  optimization,  and  waste  elimination  (Goldsby,  2005). 

Business  Process  Reengineering 

Reengineering  is  defined  as  the  fundamental  rethink  and  radical  redesign  of 
business  processes  to  generate  dramatic  improvements  in  critical  performance  measures  - 
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-  such  as  cost,  quality,  service  and  speed.  The  basic  idea  is  to  start  with  a  blank  sheet  of 
paper,  forget  about  current  processes  and  traditions,  take  what  is  known  about  customers 
and  their  preferences,  and  develop  completely  new  “optimized”  business  processes.  The 
goal  is  to  produce  simplified  business  processes,  empowered  personnel,  and  a  shift  of 
emphasis  from  individual  to  team  achievements  (Hammer,  2003). 

Theory  of  Constraints 

The  Theory  of  Constraints  (TOC),  developed  by  Dr.  Eliyahu  M.  Goldratt,  is  based 
on  the  idea  that  quality  and  productivity  will  increase  if  various  constraints  are  removed. 
The  philosophy  emphasizes  that  a  single  constraint  or  bottleneck  exists  in  any  process 
and  controls  the  output  from  the  entire  process.  A  constraint  is  anything  that  hinders  an 
organization  in  reaching  its  goals.  There  are  two  types  of  constraints:  physical  and  non¬ 
physical.  Some  examples  of  physical  constraints  are  warehouse  space,  machine  capacity, 
or  number  of  delivery  vehicles.  Some  examples  of  non-physical  constraints  are 
employee  attitudes,  customer  demand,  or  company  procedures.  Each  type  of  constraint 
must  be  identified,  categorized,  and  treated  accordingly  in  order  to  manage  performance. 

TOC  can  be  applied  to  manufacturing  processes,  such  as  inefficient  factory 
layouts,  wrong  quantity  or  type  of  inventory,  or  schedule  problems,  and  to  management 
processes,  such  as  outdated  policies  or  procedures  (Goldratt,  1992).  TOC  uses  a  six-step 
process  to  enable  ongoing  improvement: 

1 .  Identify  the  system’s  constraints 

2.  Decide  how  to  exploit  the  system’s  constraints 

3.  Subordinate  everything  else  to  the  decision  in  step  2 

4.  Elevate  the  system’s  constraints 

5.  If  the  system’s  constraints  were  changed,  return  to  step  1 

6.  Change  the  system  if  required 
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TOC  is  not  a  substitute  for  other  CPI  tools;  it  should  be  used  in  conjunction  with  other 
CPI  tools  to  reap  maximum  benefits.  Although  each  of  the  CPI  methodologies  has 
different  tool  sets  and  different  goals,  there  is  a  common  thread  —  all  involve  reducing  or 
removing  barriers  to  customer  service.  Six  Sigma  reduces  variation,  Lean  reduces  waste, 
and  TOC  reduces  constraints.  Table  1  below,  created  by  the  Defense  Acquisition 
University  (DAU)  for  CPI  familiarization  training,  depicts  a  comparison  of  each  of  these 
tools. 


Table  1.  Comparison  of  Six  Sigma,  Lean,  and  Theory  of  Constraints  (DAU,  2006) 


Program 

Six  Sigma 

Lean  Thinking 

Theory  of  Constraints 

Theory 

Reduce  Variation 

Remove  Waste 

Manage  Constraints 

Application 

Guidelines 

1 .  Define 

2.  Measure 

3.  Analyze 

4.  Improve 

5  Control 

1.  Identify  Value 

2.  Identify  Value 
Stream 

3.  Flow 

4.  Pull 

5.  Perfection 

1 .  Identify  Constraint 

2.  Exploit  Constraint 

3.  Subordinate  Process 

4.  Elevate  Constraint 

5.  Repeat  Cycle 

Focus 

Problem  Focused 

Flow  Focused 

Systems  Constraints 

Assumptions 

A  problem  exists. 

Figures  and  numbers 
are  valued. 

System  output 
improves  if 
variation  in  all 
processes  is 
reduced. 

Waste  removal  will 
improve  business 
performance. 

Many  small 
improvements 
are  better  than 
systems  ana  lysis. 

Emphasis  on  speed 
and  volume. 

Uses  existing  systems. 

Process  interdependence. 

Primary 

Effect 

Uniform  process 
output 

Reduced  flow  time 

Fast  throughput 

Secondary 

Effects 

Less  variation. 

Uniform  output. 

Less  inventory. 

Fluctuation  - 
performance 
measures  for 
managers. 

Improved  quality. 

Less  waste. 

Fast  throughput. 
Flow  -  performance 
measure  for 
managers. 
Improved  quality. 

Less  inventory/waste. 
Throughput  cost 
accounting. 

Throughput  -  performance 
measurement  system. 
Improved  quality. 

Business  Case  Analysis 


In  the  context  of  this  paper,  a  business  case  is  a  presentation  that  captures  the 


reasoning  for  initiating  a  process  improvement  project.  It  is  not  a  CPI  methodology 
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itself.  A  business  case  captures  various  characteristics  of  a  proposed  process 
improvement  project,  such  as  the  background  of  the  project,  expected  benefits,  expected 
costs,  the  alternatives  considered,  the  expected  risks  associated  with  each  alternative,  and 
the  implementation  timeline.  A  business  case  analysis  is  “a  systematic  examination  of 
alternatives  resulting  in  a  recommendation  based  on  the  ‘corporate’  good”  (AFSO  21 
Playbook,  2007).  The  main  goal  is  to  help  management  make  an  informed  decision 
whether  a  proposed  process  improvement  effort  should  be  undertaken. 

Change  Management 

Similarly,  change  management  is  not  a  CPI  methodology  in  itself,  but  it  is  a 
necessary  function  in  order  to  successfully  employ  any  major  process  improvement 
effort.  Organizational  change  management  refers  to  the  processes  and  tools  used  for 
managing  the  human  side  of  corporate  change.  According  to  Brien  Palmer  in  his  book 
Making  Change  Work:  Practical  Tools  for  Overcoming  Human  Resistance,  changes  that 
fail  usually  do  not  fail  because  of  technical  reasons  -  something  inherently  flawed  about 
the  change  itself  (2003).  They  usually  fail  because  of  human  reasons;  i.e.,  the  promoters 
of  the  change  did  not  attend  to  the  healthy,  real,  and  predictable  reactions  of  nonnal 
people  to  disturbances  in  their  routines.  Overcoming  these  human  barriers  requires  a 
special  kind  of  leader,  or  a  change  agent,  who  has  the  clout,  the  conviction,  and  the 
charisma  to  make  things  happen  (Womack,  2003).  Effective  change  management  is  a 
critical  element  of  any  process  improvement  activity. 
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The  Roll-up:  Air  Force  Smart  Operations  for  the  21st  Century 


AFS02 1  is  an  improvement  model  customized  to  the  unique  environment  of  the 
United  States  Air  Force  which  leverages  improvement  methods  from  various  sources, 
such  as  Lean,  Six  Sigma,  Theory  of  Constraints  and  Business  Process  Reengineering  with 
the  ultimate  objective  of  improving  combat  capability  (AFS021  Playbook,  2007). 
AFS021  uses  portions  of  these  methods  to  increase  operational  support,  kill  non-value- 
added  work,  use  “clean  sheet”  thinking,  improve  how  we  operate,  and  identify  gaps 
within  Air  Force  operations  to  maximize  value  and  eliminate  waste.  The  desired  effects 
of  these  Smart  Operations  are  to  increase  productivity  of  Air  Force  personnel,  increase 
critical  asset  availability,  improve  response  time  and  decision  making  agility,  sustain  safe 
and  reliable  operations,  and  improve  energy  efficiency. 

According  to  the  Director  of  ACC/A9,  AFS02 1  is  comprised  of  four  components: 
CPI,  performance  measurement,  the  8-step  problem  solving  process,  and  Strategic 
Alignment  and  Deployment  (SA&D).  The  first  component,  CPI,  has  already  been 
discussed.  The  focus  of  the  second  component,  performance  measurement,  is  to  ensure 
that  the  right  types  of  performance  (i.e.,  results  vice  activity)  are  being  measured  the  right 
way  (i.e.,  using  effects-based  data  to  track  specific  progress  from  a  known  starting  point 
to  a  new  end-state).  This  is  referred  to  as  a  “baseline-to-target”  approach.  The  third 
component  consists  of  the  following  eight-step  problem-solving  approach: 

1 .  Clarify  The  Problem 

2.  Break  Down  The  Problem/Identify  Perfonnance  Gaps 

3.  Set  Improvement  Target 

4.  Determine  Root  Causes 

5.  Develop  Countermeasures 
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6.  See  Countermeasures  Through 

7.  Confirm  Results  and  Process 

8.  Standardize  Successful  Processes 

The  first  step  is  critical  because  the  team  must  understand  which  problem  needs  to  be 
solved  in  order  to  be  in  the  best  position  to  solve  it.  The  second  step  involves  gathering 
objective  data  and  reviewing  appropriate  metrics  in  order  to  determine  the  root  cause  of 
the  problem.  The  third  step  entails  setting  a  specific,  measurable,  attainable  goal  for  the 
desired  effect  of  the  process  improvement  effort.  The  purpose  of  the  fourth  step  is  to 
identify  and  attack  the  source  of  the  problem,  as  opposed  to  performing  “quick  fixes”  on 
the  symptoms  of  the  problem.  This  will  ensure  the  problem  doesn’t  occur  again  in  the 
future.  The  purpose  of  step  five  is  to  come  up  with  a  quality  solution  that  is  practical, 
effective,  and  accepted  by  the  stakeholders  affected  by  its  implementation.  The  purpose 
of  step  six  is  to  implement  the  plan  developed  in  the  previous  steps.  Step  seven  entails  a 
comparison  of  the  metrics  of  the  new  process  with  the  old  process  to  determine  if  the 
improvement  effort  was  a  success.  The  purpose  of  step  eight  is  to  ensure  the 
implementation  of  the  new  process  is  standardized  and  that  the  results  of  the  new  process 
stick. 

The  final  aspect  of  AFS021,  Strategic  Alignment  &  Deployment  (SA&D),  refers 
to  the  DoD  methodology  for  aligning  CPI  efforts  with  strategic  policy.  SA&D  is  the 
process  of  cascading  or  communicating  policy  from  top  to  middle  management  and 
throughout  the  rest  of  the  organization  using  a  give-and-take  process  called  “catchball”. 
According  to  DoD  Instruction  5010.43,  Implementation  of  DoD-wide  CPI/Lean  Six 
Sigma,  “DoD  components  must  identify  organizational  mission,  priorities,  and  goals  from 
top  to  bottom  within  an  enterprise”  and  “leaders  shall  apply  accepted  CPI  concepts 
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through  a  disciplined  deployment  approach  that  is  focused  on  the  alignment  of  goals  and 
priorities  throughout  the  organization”.  The  purposes  of  SA&D  are  to  execute  strategy, 
maintain  focus  on  objectives,  and  measure  performance  on  meeting  those  objectives 
along  the  way  (DoDI  5010.43,  2009).  Figure  5  shows  the  strategic  policy  deployment 
model  for  CPI  in  the  DoD. 


Strategic  Deployment 


Figure  5.  DoD  Strategic  Deployment  Model  (DoD  CPI  Guidebook,  2008) 


The  first  step  of  SA&D  is  to  review  the  organization’s  mission,  priorities  and 
goals.  The  second  step  is  to  deploy  those  objectives  down  to  the  lowest  level  in  the  chain 
of  command.  The  third  step  is  to  establish  metrics  for  achieving  those  objectives  at  each 
level  in  the  chain  in  order  to  quantify  how  well  the  organization  is  meeting  its  objectives 
at  all  levels.  The  fourth  step  is  to  establish  a  perfonnance  baseline  against  those 
objectives  using  the  appropriate  metrics.  The  fifth  step  is  to  examine  that  baseline  and 
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identify  improvement  opportunities  for  closing  performance  gaps.  The  sixth  step  is  to  do 
a  CPI  event  on  the  appropriate  improvement  target  area.  The  final  step  is  to  collect  and 
review  data  after  the  improvement  effort  to  determine  its  effects  on  the  organization’s 
mission  (2009).  This  paper  proposes  a  methodology  for  implementing  steps  three 
through  five  of  SA&D  at  the  enterprise  level  of  an  organization. 

What  AFS021  Is  Not 

AFS021  is  not  a  substitute  for  commanders’  responsibilities  at  all  levels.  It  is  not 
a  substitute  for  improved  Air  Force  corporate  management  processes.  And  it  is  not 
another  fad  for  process  improvement  like  Total  Quality  Management  (TQM)  that  is  here 
today  and  gone  tomorrow  (AFS021  Playbook,  2007).  With  the  right  type  of  leadership 
and  buy-in  from  individuals  at  the  middle  management  and  working  levels,  AFS02 1  is  a 
systematic  methodology  that  has  the  potential  to  help  organizational  units  streamline 
internal  processes,  better  utilize  their  workforce,  and  produce  higher  quality  products 
more  efficiently. 

Metric  Development 

Measurement  Basics 

A  metric  is  defined  as  a  standard  of  measurement  (Miriam- Webster),  where 
measurement  is  the  objective  representation  of  objects,  processes,  and  phenomenon 
(Finkelstein,  1984).  Measurement  captures  infonnation  about  systems  through  their 
attributes,  which  can  be  either  directly  or  indirectly  observable  (Cropley,  1998). 

All  measurement  is  done  within  a  context  (Morse,  2003),  which  is  shaped  by  a 
purpose,  existing  knowledge,  capabilities,  and  resources  (Brakel,  1984).  Within  this 
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context,  measurement  begins  by  identifying  the  system  of  interest  and  the  attributes  to  be 
used  in  defining  the  system.  In  the  context  of  business,  a  metric  is  a  measure  used  to 
evaluate  an  organization’s  progress  toward  its  goals  (Kirkwood,  1997).  Attribute 
selection  is  important  because  the  validity  of  a  system  measurement  is  influenced  by  the 
number  of  attributes  used  in  the  measurement.  If  the  wrong  system  attributes  are  chosen, 
the  perceived  state  of  a  system  may  be  different  from  the  actual  state,  so  this  is  a  key 
consideration  of  any  framework  for  effectiveness  measurement  (Potter,  2000).  In 
addition  to  the  type  of  attributes  chosen,  the  number  of  attributes  selected  must  be 
considered.  Fewer  attributes  will  simplify  the  measurement  process,  but  too  few  can 
result  in  poor  and/or  misleading  insights  about  the  system  (Sink,  1985).  Once  the 
appropriate  set  of  attributes  is  identified,  data  collection  on  the  system  can  take  place. 

Types  of  Scales 

The  most  common  scale  types  are  the  Nominal,  Ordinal,  Interval,  Ratio,  and 
Absolute  scales  (Sarle,  1995).  A  nominal  scale  only  contains  equivalence  meaning.  The 
ordinal  type  has  both  equivalence  and  rank  order  meaning.  Interval  measures  have  these 
two  meanings  as  well,  but  also  have  meaning  in  the  intervals  between  the  values.  Ratio 
measurement  further  adds  meaning  in  the  ratios  of  values.  Finally,  absolute  scales 
measure  ratios  without  units  (Ford,  1993).  Figure  6  shows  the  relationships  between 
these  scales. 
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Figure  6.  Scale  Hierarchy  of  Commonly  Used  Measures  (Ford,  1993) 

Measurement  scales  can  be  further  categorized  as  either  natural  or  constructed, 
and  as  direct  or  proxy.  A  natural  scale  is  one  that  is  accepted,  understood,  and  used  by 
everyone.  For  example,  profit  in  dollars  is  a  natural  scale  that  is  used  for  many  business 
decisions.  A  constructed  scale  is  one  that  must  be  created  for  a  particular  type  of 
problem  because  there  is  none  that  exists  naturally  to  measure  how  well  an  objective  that 
is  specific  to  that  problem  is  achieved.  For  example,  a  teacher  uses  a  constructed  scale  of 
letter  grades  to  measure  students’  comprehension  of  concepts  they  have  been  taught.  In 
this  case,  the  letter  grades  A,  B,  C,  D,  and  F  represent  scores  which  are  >90,  >80,  >70, 
>60,  and  <60,  respectively.  A  direct  scale  directly  measures  the  degree  of  attainment  of 
an  objective,  whereas  a  proxy  scale  measures  an  associated  objective,  i.e.  one  that  acts  as 
a  substitute  (Kirkwood,  1997).  A  proxy  measure  is  essentially  a  model  or  approximation 
of  the  system  attribute  of  interest  (Potter,  2000).  For  example,  the  number  of  dollars  is  a 
direct  scale  used  to  measure  “profit”,  whereas  gross  national  product  is  a  proxy 
(substitute)  scale  for  measuring  “the  economic  well-being  of  the  country”.  These  two 
distinctions  actually  represent  the  range  of  scale  types  that  can  be  used,  from  the  previous 
example  of  profit,  which  uses  a  natural  direct  scale  to  measure  the  amount  of  money 
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earned,  to  class  grades,  by  which  professors  use  a  constructed  proxy  scale  to  measure  a 
person’s  intelligence  (Kirkwood,  1997).  Other  examples  can  be  found  in  Table  2. 


Table  2.  Measure  Types  (Kirkwood,  1997) 


Natural 

Constructed 

Direct 

Commonly  understood  measures  directly 
linked  to  strategic  objective 

Measures  directly  linked  to  the  strategic 
objective  but  developed  for  a  specific  purpose 

Example:  Profit 

Example:  Gymnastics  scoring 

Proxy 

In  general  use  measures  focused  on  an 
objective  correlated  with  the  strategic 
objective 

Measures  developed  for  a  specific  purpose 
focused  on  an  objective  correlated  to  the 
strategic  objective 

Example:  GNP  (economic  well  being) 

Example:  Student  grades  (intelligence) 

MOEs  and  MOPs 

Measurement  is  fundamental  to  understanding,  controlling,  and  forecasting 
(Wilbur,  1995).  Measures  of  effectiveness  (MOEs)  and  measures  of  performance 
(MOPs)  provide  different  insights  about  a  system.  A  MOE  concerns  how  well  a  system 
tracks  against  its  purpose  (Sproles,  1997).  However,  a  MOP  describes  how  well  a  system 
uses  resources  (Sink,  1985).  In  other  words,  a  MOE  detennines  if  the  right  things  are 
being  done,  and  a  MOP  detennines  if  things  are  being  done  right  (Sproles,  1997).  The 
key  distinction  between  the  two  is  that  a  MOP  measures  activity,  but  it  does  not  indicate  a 
system’s  progress  toward  its  purpose. 


Key  Performance  Indicators  (KPIs) 

Many  terms  are  often  used  to  describe  the  result  of  an  observation,  such  as 
measurement,  indicator,  or  metric.  In  the  context  of  business,  key  perfonnance  indicators 
(KPIs)  are  the  few  vital  statistics  that  indicate  the  health  of  an  organization.  They  are  the 
quantifiable  measures  of  effectiveness,  efficiency,  and  quality  which  reflect  the 
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performance  of  an  organization  in  achieving  predefined  strategic  goals  and  objectives. 
KPIs  measure  effectiveness  by  determining  if  the  organization  has  done  what  they  said 
they  would  do,  efficiency  by  comparing  how  many  resources  were  actually  used  to  how 
many  were  planned  to  be  used,  and  quality  by  gauging  if  the  planned  efforts  were  done 
well  (Sink/Tuttle,  1989). 

There  are  two  types  of  indicators:  lagging  and  leading.  Lagging  indicators,  such 
as  most  financial  metrics,  measure  the  output  of  past  activity,  i.e.,  they  are  collected  and 
reported  after  the  fact.  Lagging  indicators  are  useful  for  tracking  trends,  but  by  the  time 
the  trend  is  noticed,  there  may  already  be  problems.  Leading  (outcome-based)  indicators 
measure  key  drivers  of  business  value  and  can  be  used  as  future  performance  drivers  for 
an  organization  (Kaplan/Norton,  1996).  KPIs  are  the  critical  drivers  for  mission  success. 
KPIs  are  metrics,  but  not  all  metrics  are  KPIs. 

Traps  of  Metrics 

There  are  literally  thousands  of  metrics  to  choose  from;  the  difficulty  lies  in 
identifying  the  right  ones.  It  is  easy  to  use  metrics  that  do  not  measure  the  aspects  that 
pertain  to  an  organization’s  specific  problem.  Another  issue  is  “paralysis  by  analysis”. 
When  data  is  not  tied  to  the  organization’s  KPIs,  employees  end  up  spending  a  lot  of  time 
collecting  and  analyzing  data  that  doesn’t  impact  the  bottom  line  (Kaplan/Norton,  1996). 
A  specific  instance  of  this  occurs  when  an  organization  uses  activity  (input)  metrics 
versus  results-focused  (output)  metrics.  Basically,  they  end  up  reporting  status  on  the 
work  they’re  doing  to  meet  improvement  goals,  instead  of  reporting  on  the  organization’s 
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actual  progress  toward  meeting  its  goals  (Potter,  2000).  Creating  metrics  that  have  the 
following  characteristics  will  help  organizations  avoid  running  into  these  and  other  traps. 


Characteristics  of  Effective  Metrics 

The  key  to  successful  measurement  is  ensuring  the  right  measures  are  being  used 
to  gauge  the  system’s  purpose  (Brown,  1996).  Effective  metrics  share  some  of  the 
following  characteristics:  strategically-linked,  timely,  objective,  complete,  obtainable, 
and  valid. 

•  Strategically-linked  -  Effectiveness  measures  should  be  traceable  to  the 
organization’s  strategic  purpose  (Kaplan,  1991).  Metrics  should  be  specific  and 
targeted  to  the  organization’s  KPIs  to  ensure  they  only  measure  the  outcomes  that 
have  value  to  the  customer  (Kaplan/Norton,  1996). 

•  Timely  -  Measures  should  be  collected  and  processed  in  a  timeframe  that  is 
needed  to  be  relevant  within  the  context  (Kaplan,  1991). 

•  Objective  -  Measures  should  be  easy  to  understand.  It  should  be  clear  when  you 
chart  your  performance  over  time  which  direction  is  "good"  and  which  direction 
is  "bad",  so  that  you  know  when  to  take  action  (Finkelstein,  2003). 

•  Complete  -  Measures  should  address  all  KPIs  in  enough  detail  to  accurately 
depict  the  status  of  the  key  mission  areas  of  the  organization  (Kaplan,  1991).  The 
number  of  measures  should  be  limited  in  order  to  avoid  information  overload. 

•  Obtainable  -  Measures  should  be  readily  and  easily  obtainable  from  available 
sources  (Keeney,  1992). 

• 

•  Valid  -  Measures  should  actually  measure  what  is  intended  to  be  measured  in 
order  to  be  meaningful  (Carton,  2006). 

Displaying  Metrics 

Finally,  an  important,  but  often  underemphasized  aspect  of  system  measurement 
is  communicating  the  information.  The  goal  for  an  information  display  should  be  to 
present  the  maximum  amount  of  infonnation  possible  while  ensuring  unambiguous 
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understanding  of  the  insights  it  presents  (Tufte,  1997).  An  increasingly  popular  method 
of  displaying  information  is  through  the  use  of  a  metrics  dashboard  (Eckerson,  2006). 

Unfortunately,  there  is  no  comprehensive  method  for  developing  a  complete  set  of 
measures.  However,  achieving  completeness  typically  requires  both  critical  and  creative 
thinking  in  an  iterative  process  involving  negotiation  and  compromise  among  those 
interested  in  and  knowledgeable  about  the  system  (Sproles,  2002). 

Linear  Regression 

According  to  Kutner,  Nachtsheim,  Neter,  and  Li,  “regression  analysis  is  a 
statistical  methodology  that  utilizes  the  relation  between  two  or  more  quantitative 
variables  so  that  a  response  or  outcome  variable  can  be  predicted  from  the  other,  or 
others”  (Kutner,  et  al,  2005). 

Regression  is  used  in  a  variety  of  disciplines,  from  business  to  behavioral 
sciences,  for  three  major  purposes:  description,  control,  and  prediction.  When 
considering  the  concept  of  a  relation  between  variables,  Kutner  highlights  the  importance 
of  distinguishing  between  a  functional  relation  and  a  statistical  relation.  A  functional 
relation  between  two  variables  is  expressed  by  a  mathematical  fonnula.  For  example,  if 
Xi,  X2,  X3,  and  X4  denote  the  independent  variables  and  Y  the  dependent  variable,  a 
functional  relation  is  of  the  form: 

Y=/(X!,X2,X3,X4) 

Given  a  particular  value  of  each  X,  the  function/ indicates  the  corresponding  value  of  Y. 
Figure  7  shows  an  example  of  a  functional  relation  between  the  predictor  variable  “Units 
Sold”  and  the  single  response  variable  “Dollar  Sales”. 
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Figure  7.  Example  of  Functional  Relation  (Kutner,  2005) 

A  statistical  relation,  on  the  other  hand,  is  not  perfect  like  a  functional  relation, 
because  observations  for  a  statistical  relation  generally  do  not  fall  directly  on  the  curve  of 
relationship.  Figure  8  below  shows  an  example  of  a  statistical  relation  between  the 
predictor  variable  “Midyear  Evaluation”  and  the  response  variable  “Year-End 
Evaluation”. 


Midyear  Evaluation  Midyear  Evaluation 


Figure  8.  Example  of  Statistical  Relation  (Kutner,  2005) 

The  coefficient  of  determination,  R~,  is  a  measure  that  may  be  interpreted  as  the 
proportionate  reduction  of  total  variation  associated  with  the  use  of  the  predictor  variable 
X  (where  0  <  R  <  1).  Thus,  the  larger  R  is,  the  more  the  total  variation  of  Y  is  reduced 
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by  introducing  the  predictor  variable  X  (Kutner,  2005).  Basically,  R"  indicates  how  well 
a  regression  model  describes  the  relation  between  the  observed  values  of  X  and  Y. 

The  correlation  coefficient,  r,  represents  the  measure  of  linear  association 
between  Y  and  X  when  both  Y  and  X  are  random.  It  can  vary  from  0  (which  indicates  no 
correlation)  to  ±1  (which  indicated  perfect  correlation).  When  the  correlation  coefficient 
is  greater  than  0,  the  two  variables  are  said  to  be  positively  correlated;  i.e.,  when  one  is 
large,  the  other  is  also  large.  When  it  is  less  than  0,  they  are  said  to  be  negatively 
correlated;  i.e.,  when  one  is  large,  the  other  is  small  (Makridakis,  1998). 

Case  Study:  Air  Combat  Command  (ACC) 

ACC  is  the  primary  force  provider  of  combat  airpower  to  America's  warfighting 
commands.  ACC  operates  fighter,  bomber,  reconnaissance,  battle-management,  and 
electronic-combat  aircraft  to  support  global  implementation  of  a  national  security 
strategy.  ACC  also  provides  command,  control,  communications  and  intelligence 
systems,  and  conducts  global  infonnation  operations.  In  addition,  ACC  “organizes, 
trains,  equips  and  maintains  combat-ready  forces  for  rapid  deployment  and  employment 
while  ensuring  strategic  air  defense  forces  are  ready  to  meet  the  challenges  of  peacetime 
air  sovereignty  and  wartime  air  defense”  (www.my.af.mil). 

The  Commander  of  ACC  (COMACC)  has  two  roles:  one  is  to  organize,  train,  and 
equip  the  forces  within  ACC,  and  the  other  is  to  direct  the  actions  of  the  Combat  Air 
Force  (CAF).  The  CAF  is  a  collection  of  Air  Force  organizations,  commands  and  forces 
tasked  to  generate  specific  precise  effects  from  the  air,  space,  and  cyberspace.  The 
CAF’s  mission  is  to  “Fly,  Fight,  and  Win  —  integrating  capabilities  across  air,  space,  and 
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cyberspace  to  deliver  precise  coercive  effects  in  defense  of  our  Nation  and  its  global 
interests”  (CAF  Strategic  Plan  2010). 

The  ACC/A9  AFS021  office  manages  the  CPI  program  for  Headquarters  ACC. 
According  to  the  Director  of  ACC/A9,  “Excellence  in  all  we  do  directs  us  to  develop  a 
sustained  passion  for  the  continuous  improvement  and  innovation  that  will  propel  the  Air 
Force  into  a  long-term,  upward  spiral  of  accomplishment  and  performance.” 
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III.  Methodology 


This  chapter  presents  a  methodology  for  developing  metrics  to  identify  how  and 
where  to  use  continuous  process  improvement  (CPI)  in  an  organization.  Each  step  will 
be  described  for  the  general  case,  followed  by  an  example  of  a  specific  application  to  the 
ACC  case  study. 

In  order  for  an  organization  to  implement  CPI  methods  effectively,  it  must  have 
the  right  objective  from  the  outset.  Contrary  to  popular  belief,  the  ultimate  goal  of  CPI  is 
not  to  simply  save  money  or  reduce  man-hours  (though  these  are  certainly  positive  side 
effects).  No,  for  an  organization  to  achieve  success  it  must  go  much  deeper  than  these 
“symptoms”  and  examine  its  sole  purpose  for  existence:  it  must  identify  its  bottom  line. 
In  addition  to  identifying  the  right  objective  at  the  beginning,  CPI  must  be  aligned  with 
the  strategic  priorities  of  the  organization’s  leadership  (Sink/Tuttle,  1989).  In  chapter  2, 
how  this  is  done  in  the  DoD  via  the  SA&D  process  is  discussed. 

Arguably,  the  sole  purpose  for  existence  for  any  company  is  to  make  a  profit. 
Consider  the  previous  examples  of  saving  money  and  reducing  man-hours.  The  reason 
for  saving  money  (i.e.,  reducing  operating  costs)  is  to  increase  profit.  Likewise,  the 
reason  for  cutting  man-hours  (i.e.,  reducing  labor  costs),  is  to  increase  profit.  Therefore, 
for  a  CPI  event  to  have  the  most  success  with  lasting  results  for  a  company,  the  target  for 
improvement  must  be  the  bottom  line,  which  is  to  increase  profit. 

The  military,  however,  does  not  work  for  profit,  so  how  do  we  define  our  bottom 
line?  According  to  Sink  and  Tuttle,  improving  performance  is  the  bottom  line 
(Sink/Tuttle,  1989).  The  current  mission  of  the  US  Air  Force  is  “to  fly,  fight  and  win. . . 
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in  Air,  Space,  and  Cyberspace”  (Air  Force  Fact  Sheet,  2010).  Our  primary  shareholders, 
the  American  public,  expect  us  to  perform  our  mission  well  in  order  to  protect  our  nation. 
To  that  end,  the  Air  Force’s  bottom  line  is  our  warfighting  capability.  It’s  just  not  about 
“doing  more  with  less”.  It’s  about  doing  what  we  do  better. 

As  noted  previously,  ACC’s  mission  is  to  provide  forces  of  combat  airpower  to 
America's  warfighting  commands.  Therefore,  ACC’s  bottom  line  is  readiness  to  provide 
warfighting  capability.  According  to  DoD  Directive  7730.65,  readiness  is  defined  as  “a 
measure  of  the  Department  of  Defense's  ability  to  provide  the  capabilities  needed  to 
execute  the  missions  specified  in  the  National  Military  Strategy”  (2007).  The  Combat 
Air  Force  (CAF)  Strategic  Plan  states  that  winning  the  current  fight  and 
deterring/winning  the  future  fight  requires  “a  credible,  ready  force”,  and  the  decision 
maker  for  ACC,  the  COMACC  will  “place  increased  emphasis  on  readiness ”  (2010, 
emphasis  added).  Therefore,  the  COMACC  values  readiness  as  the  critical  indicator  for 
his  organization’s  ability  to  perfonn  its  mission. 

Now  that  the  decision  maker’s  values  have  been  detennined,  the  next  step  is  to 
choose  appropriate  KPIs  that  can  be  used  to  signify  how  well  the  organization  is  meeting 
its  bottom  line.  For  the  ACC  case  study,  the  primary  KPI,  readiness,  can  be  measured 
using  ratings  from  the  Status  of  Resources  and  Training  System  (SORTS),  Operational 
Readiness  Inspection  (ORI)  reports,  the  Defense  Readiness  Reporting  System  (DRRS), 
and  the  AEF  (Air  Expeditionary  Force)  Unit  Type  Code  (UTC)  Status  Reporting  Tool 
(ART).  AFI  10-201  SORTS  states  that, 

“SORTS  is  the  single  automated  reporting  system  within 
the  Department  of  Defense  (DoD)  functioning  as  the 
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central  registry  of  all  operational  units  of  the  US  Armed 
Forces  and  certain  foreign  organizations.  SORTS  has  a 
threefold  purpose:  it  provides  data  critical  to  crisis 
planning,  provides  for  the  deliberate  or  peacetime  planning 
process,  and  is  used  by  the  Chief  of  Staff  United  States  Air 
Force  (CSAF)  and  subordinate  commanders  in  assessing 
their  effectiveness  in  meeting  Title  10,  “United  States 
Code,”  responsibilities  to  organize,  train,  and  equip  forces 
for  combatant  commands.  The  Air  Force  uses  SORTS 
status  infonnation  in  assessing  readiness,  analyzing 
readiness  trends,  and  supporting  readiness  decisions. 

SORTS  provides  broad  bands  of  infonnation  on  selected 
unit  status  indicators  which  include  the  commander’s 
assessment  of  the  unit’s  ability  to  execute  the  mission  set 
for  which  it  is  organized  or  designed  (AFI  10-201,  2006, 
emphasis  added)”. 

In  addition  to  SORTS  reports,  AFPD  10-2  AF  Readiness  dictates  that  “the  Air 
Force  will  continually  assess  readiness  based  on  criteria  established  by  the  Secretary  of 
Defense,  CJCS,  Secretary  of  the  Air  Force  and  the  Chief  of  Staff  of  the  Air  Force. 
Training,  exercise  evaluation  and  inspection  assessments  are  critical  and  will  be  used  for 
readiness  assessments”  (AFPD  10-2,  2006).  These  assessments  are  called  ORIs,  which 
are  described  in  AFI  90-201  Inspector  General  Activities  below: 


“ORIs  evaluate  and  measure  the  ability  of  units  to  perfonn 
their  wartime,  contingency,  or  force  sustainment  missions. 

Phase  I  evaluates  the  unit’s  transition  from  peacetime 
readiness  and  the  unit's  ability  to  maintain  and  sustain 
essential  home  station  missions  during  and  after  the 
deployment  of  forces  and  includes  the  major  graded  area 
(MGA)  of  Positioning  the  Force.  Phase  II  evaluates  the 
unit’s  ability  to  meet  wartime/contingency  taskings  through 
the  MGAs  of  Employing  the  Force,  Sustaining  the  Force, 
and  Ability  to  Survive  and  Operate  (ATSO)”. 

AFI  10-244  Reporting  Status  of  AEFs  describes  the  AEF  as  “the  Air  Force’s 
methodology  for  organizing,  training,  equipping,  and  sustaining  rapidly  responsive  air 
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and  space  forces  to  meet  defense  strategy  requirements.  Through  the  AEF,  the  Air  Force 
supports  defense  strategy  requirements  using  a  combination  of  both  permanently  assigned 
and  rotational  (allocated)  forces”  (2008). 

Readiness  can  also  be  measured  using  the  DRRS,  which  is  described  in  AFPD  10- 
2  AF  Readiness  as  “an  OSD  net-centric,  web-enabled  initiative  to  manage  and  report  the 
readiness  of  DOD  forces  to  execute  the  National  Military  Strategy  (NMS).  All  Air  Force 
readiness  related  programs  and  processes  will  be  aligned  with  DRRS  initiatives”  (2006, 
emphasis  added).  According  to  DoD  Directive  7730.65  DRRS ,  this  system  is  used  to 
measures  and  report  on  the  readiness  of  military  forces  and  the  supporting  infrastructure 
to  meet  missions  and  goals  assigned  by  the  Secretary  of  Defense.  DRRS  is  “a 
capabilities-based,  adaptive,  near  real-time  readiness  reporting  system  for  the 
Department  of  Defense”  (2007,  emphasis  added).  The  DRRS  User  Guide  says  that 
DRRS  “assesses  a  deployed  unit’s  ability  to  perfonn  its  mission  essential  tasks  while 
assigned  to  a  Combatant  Commander  (COCOM).  The  Air  Force  identifies  Unit  Type 
Code  (UTC)  elements  that  can  be  deployed  separately  or  as  part  of  a  capability  package 
to  support  a  designated  operation”  (2010). 

In  addition  to  DRRS,  the  AEF  Reporting  Tool  (ART)  is  also  used  to  assess 

deployment  readiness,  which  is  introduced  in  AFPD  10-2  Readiness: 

“the  ART  allows  AEF  allocated  units  the  ability  to  report 
UTC  level  readiness  data.  It  allows  immediate  updates  and 
ready  access  to  an  aggregate  UTC  status  for  all  levels  of 
command  with  sufficient  depth  of  information  to  make 
informed  decisions  on  the  employment  of  Air  Force 
capabilities  across  the  full  range  of  military  operations. 

Integration  of  DRRS  and  ART  is  critical  to  provide  the 
required  visibility  of  Air  Force  capabilities  and  resources 
while  supporting  the  AEF  construct.” 
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ART  measures  the  unit’s  ability  to  meet  the  Combatant  Commander’s  (COCOM) 
requirements  outlined  in  operational  plans. 

Once  KPIs  have  been  chosen,  the  next  step  is  to  solicit  metrics  and  success 
criteria  from  the  decision  maker  to  determine  how  the  organization  is  performing.  Per 
AFI  10-201,  SORTS  documents  and  measures  four  categories  that  detennine  the  unit’s 
ability  (readiness)  to  perfonn  its  mission  based  on  the  unit's  full  wartime  requirement  for 
which  it  was  organized  or  designed: 

1 .  Personnel  -  This  refers  to  the  number  of  personnel  assigned, 
authorized,  and  available  to  perfonn  the  unit’s  mission  based 
on  Unit  Manning  Document  or  UTC  requirements. 

This  category  is  denoted  by  the  letter  “P”  on  SORTS  reports. 

2.  Training  -  This  refers  to  the  required  training  for  personnel 
assigned  to  the  unit  to  perform  the  mission. 

This  category  is  denoted  by  the  letter  “T”  on  SORTS  reports. 

3.  Equipment/Supplies  -  This  refers  to  the  number  and  type  of 
equipment  required  to  perform  the  unit’s  mission. 

This  category  is  denoted  by  the  letter  “S”  on  SORTS  reports. 

4.  Equipment  Condition  -  This  refers  to  the  condition  of 
possessed  equipment  and  supplies  required  to  perfonn  the 
unit’s  mission. 

This  category  is  denoted  by  the  letter  “R”  on  SORTS  reports. 

AFI  10-201  states  that  “category-levels  (C-levels)  reflect  the  degree  to  which  unit 
resources  meet  prescribed  levels  of  personnel,  equipment,  and  training”.  Each  of  the  four 
categories  (P,  T,  S,  and  R)  is  assigned  a  C-level  on  a  scale  of  1  to  6,  where  1  and  4 
represent  the  best  and  worst  ratings,  respectively,  and  C-levels  of  5  and  6  are  used  to 
indicate  that  the  unit  is  not  required  to  report  at  the  present  time  (C-levels  of  5  and  6  are 
not  used  as  ratings).  C-levels  are  defined  as  follows: 
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C-l.  The  unit  possesses  the  required  resources  and  is 
trained  to  undertake  the  full  wartime  mission(s)  for  which  it 
is  organized  or  designed.  The  resource  and  training  area 
status  will  neither  limit  flexibility  and  methods  for  mission 
accomplishment  nor  increase  vulnerability  of  unit 
personnel  and  equipment.  The  unit  does  not  require  any 
compensation  for  any  deficiencies. 

C-2.  The  unit  possesses  the  required  resources  and  is 
trained  to  undertake  most  of  the  wartime  mission(s)  for 
which  it  is  organized  or  designed.  The  resource  and  training 
area  status  may  cause  isolated  decreases  in  flexibility  in 
methods  for  mission  accomplishment,  but  will  not  increase 
the  unit's  vulnerability  under  most  envisioned  operational 
scenarios.  The  unit  would  require  little,  if  any, 
compensation  for  deficiencies. 

C-3.  The  unit  possesses  the  required  resources  and  is 
trained  to  undertake  many,  but  not  all,  portions  of  the 
wartime  mission(s)  for  which  it  is  organized  or  designed. 

The  resource  and  training  area  status  will  result  in 
significant  decrease  in  flexibility  for  mission 
accomplishment  and  will  increase  vulnerability  of  the  unit 
under  many,  but  not  all,  envisioned  operational  scenarios. 

The  unit  would  require  significant  compensation  for 
deficiencies. 

C-4.  The  unit  requires  additional  resources  or  training  to 
undertake  its  wartime  mission(s),  but  it  may  be  directed  to 
undertake  portions  of  its  wartime  mission(s)  with  resources 
on  hand. 

C-5.  The  unit  is  undergoing  a  Sendee-directed  resource 
action  and  is  not  prepared,  at  this  time,  to  undertake  the 
mission  set  for  which  it  is  organized  or  designed. 

C-6.  The  unit  is  not  required  to  measure  assets  in  a 
specified  area.  C-6  (not  a  rating)  may  not  be  used  as  an 
Overall  C-level. 

AFI  10-201  instructs  that  “unit  commanders  assign  the  Overall  C-level  each  time  it  is 
reported  based  on  unit  readiness.  Normally,  the  lowest  level  of  the  four  measured 
resource  areas  is  reported  as  the  Overall  C-level  provided  it  is  a  realistic  indication  of  the 
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unit’s  readiness  (based  on  the  C-level  definitions)”.  In  other  words,  the  overall  SORTS 
C-level  for  a  particular  unit  is  the  maximum  C-level  of  its  sub-categories.  For  example, 
if  the  C-levels  for  a  particular  unit  are  P-3,  T-2,  S-2,  and  R-l,  then  the  unit’s  overall 
readiness  level  is  C-3.  ACC  is  considered  “Green”  if  70%  of  its  units  are  C-l  or  C-2. 

ORIs  measure  a  wing’s  overall  readiness  “using  the  five-tier  scale  Outstanding, 
Excellent,  Satisfactory,  Marginal,  and  Unsatisfactory”,  as  presented  in  AFI  90-201.  ACC 
is  considered  “Green”  if  100%  of  its  wings  are  rated  “Satisfactory”  or  higher. 

The  DRRS  User  Guide  states  that  a  unit  that  has  been  assigned  mission  essential 
tasks  will  use  the  following  rating  structure  to  classify  its  ability  to  “accomplish  its  task 
or  mission  to  prescribed  standards  and  conditions”: 


Yes  (Green):  The  organization  can  accomplish  its  mission 
essential  task  or  mission  to  prescribed  standards  and 
conditions.  The  “Yes”  assessment  should  reflect 
demonstrated  perfonnance  in  training  or  operations. 

Qualified  Yes  (Yellow):  The  organization  can  accomplish 
most  or  all  of  the  task  to  standard  under  most  conditions. 
The  mission  essential  task  assessment  must  clearly  define 
the  specific  standard  and  conditions  that  cannot  be  met,  as 
well  as  the  shortfalls  or  issues  impacting  the  unit's  inability 
to  accomplish  the  task. 

No  (Red):  The  organization  is  unable  to  accomplish  the 
mission  essential  task  or  mission  to  prescribed  standards 
and  conditions  at  this  time. 

Not  Assessed  (Striped):  The  organization  has  not  yet  been 
assessed  (DRRS  User  Guide,  2010). 


When  all  of  the  mission  essential  tasks  assigned  to  a  unit  have  been  assessed,  that 
information  is  ready  to  be  used  towards  assessing  the  overall  mission  capabilities.  ACC 
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is  considered  “Green”  if  100%  of  deployed  units  report  “Yes”  or  “Qualified  Yes”  in 


DRRS. 


Per  AFI  10-244  Reporting  Status  of  AEFs,  the  overall  assessment  of  a  UTC  is 

rated  in  ART  using  the  following  guidelines: 

Green  =  Go.  All  identified  personnel,  equipment  and 
training  for  the  AEF  allocated  UTC  are  available  for 
deployment  within  72  hours  of  notification  or  sooner  if 
subject  to  more  stringent  criteria. 

Yellow  =  Caution.  The  UTC  has  a  missing  or  deficient 
capability;  but  that  missing  or  deficient  capability  does  not 
prevent  the  UTC  from  being  tasked  and  accomplishing  its 
mission  in  a  contingency  and/or  AEF  rotation. 

Red  =  No  Go.  The  UTC  has  a  missing  or  deficient 
capability  that  prevents  the  UTC  from  being  tasked  and 
accomplishing  its  mission  in  a  contingency  and/or  AEF 
rotation. 

ACC  is  considered  “Green”  if  100%  of  UTCs  are  Green  or  Yellow. 

Once  data  has  been  identified  for  each  KPI,  the  next  step  is  to  retrieve  that  data 
from  all  source  databases  and  combine  into  a  single  local  database.  It  will  likely  be 
necessary  to  create  a  new  database  and  adjust  the  fonnat  of  the  source  data  in  order  to 
make  a  common  format  for  all  data.  For  the  ACC  case  study,  ORI  ratings  are  available 
on  the  ACC/IG  website  on  the  Air  Force  Portal.  The  search  was  narrowed  to  include  all 
ORIs  (Phase  I,  Phase  II,  and  full  inspections)  for  Active  Duty  units  only  from  October 
2008  through  October  2010  (the  time  when  this  study  was  initiated).  Then  the  data  set 
was  reduced  by  selecting  only  aviation  wings.  This  resulted  in  12  data  points  for  ORIs 
during  the  analyzed  timeframe.  SORTS  scores  are  available  in  monthly  reports  prepared 
by  ACC/A30R.  ACC/A3  had  collected  this  data  monthly  from  the  SORTS  website  on 
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the  Secret  Internet  Protocol  Router  Network  (SIPRNet)  and  prepared  their  own 
PowerPoint  briefings  containing  the  data.  (Historical  data  was  not  available  through  the 
SORTS  website.)  DRRS  scores  are  available  through  the  DRRS  website  on  the  SIPRNet. 
Note:  DRRS  scores  were  not  used  because  the  historical  data  was  incomplete  since  all 
units  were  not  yet  required  to  report  their  scores  during  the  analyzed  timeframe.  ART 
scores  are  available  through  the  ART  website  on  the  SIPRNet.  Note:  ART  scores  were 
not  used  because  historical  data  is  not  maintained  on  the  ART  website.  A 
recommendation  to  maintain  this  data  is  included  in  Chapter  5. 

Since  each  database  has  its  own  unique  format,  a  single  data  table  was  created 
using  Microsoft  Excel  to  combine  the  data.  Table  3  depicts  this  data  table  with  notional 
data. 


Table  3.  Excel  Data  Table  with  Notional  SORTS  Scores 


A 

D 

E 

F 

G 

H 

i 

J 

K 

L 

1 

Date 

NAF  ff 

Wing  # 

Unit# 

Location 

P 

T 

S 

R 

C-level 

(max) 

2 

20- Feb-09 

5 

1 

3 

Loc  24 

3 

1 

1 

1 

3 

3 

9-Apr-09 

5 

1 

3 

Loc  24 

3 

1 

1 

1 

3 

4 

17-Jun-09 

5 

1 

3 

Loc  24 

3 

1 

1 

1 

3 

5 

20- Aug-09 

5 

1 

3 

Loc  24 

4 

1 

1 

1 

4 

6 

15- Oct-09 

5 

1 

3 

Loc  24 

4 

1 

1 

1 

4 

7 

21- Dec-09 

5 

1 

3 

Loc  24 

4 

1 

1 

1 

4 

8 

20- Feb-09 

5 

1 

4 

Loc  24 

1 

3 

1 

1 

3 

9 

15-Oct-09 

5 

1 

5 

LOC  24 

3 

1 

1 

1 

3 

10 

17- Oct-08 

5 

1 

6 

LOC  24 

2 

1 

4 

1 

4 

11 

18-Dec-08 

5 

1 

6 

Loc  24 

4 

1 

4 

1 

4 

12 

20- Feb-09 

5 

1 

6 

LOC  24 

4 

1 

4 

1 

4 

13 

9-Apr-09 

5 

1 

6 

LOC  24 

3 

1 

4 

1 

4 

14 

17-Jun-09 

5 

1 

6 

Loc  24 

3 

1 

4 

1 

4 

15 

20- Aug-09 

5 

1 

6 

LOC  24 

3 

1 

4 

1 

4 

16 

15-Oct-09 

5 

1 

6 

Loc  24 

2 

1 

4 

1 

4 

17 

21- Dec-09 

5 

1 

6 

Loc  24 

2 

1 

4 

1 

4 

18 

20- Nov-08 

5 

1 

8 

LOC  24 

4 

1 

1 

1 

4 

19 

22-Jan-09 

5 

1 

8 

LOC  24 

4 

1 

1 

1 

4 

20 

20-Mar-09 

5 

1 

8 

Loc  24 

3 

1 

1 

1 

3 

21 

16-Jul-09 

5 

1 

8 

LOC  24 

4 

1 

1 

1 

4 

22 

9-Sep-09 

5 

1 

8 

Loc  24 

3 

1 

1 

1 

3 

23 

17-Sep-10 

5 

1 

8 

Loc  24 

3 

1 

1 

4 

4 
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Once  all  of  the  KPI  scores  have  been  collected,  they  should  be  examined  for  any 
interdependencies  between  them.  The  recommended  tool  for  this  situation  is  a  linear 
regression  model  because  it  can  reveal  association  between  two  or  more  variables,  as  well 
as  predict  values  for  a  response  variable  y  based  on  a  set  of  explanatory  variables  x/ 
through  xn,  where  n  is  the  number  of  indicators  chosen  for  a  specific  situation 
(Makridakis,  1998).  For  example,  a  company  may  be  interested  in  predicting  future 
sales  for  a  particular  product  y  based  on  three  different  indicators:  past  perfonnance  of 
the  product  (xj),  inflation  rates  (jc?),  and  time  of  year  fo).  For  the  ACC  case  study,  the  P, 
T,  S,  and  R  scores  in  SORTS  (the  explanatory  variables)  were  tested  for  interaction 
between  each  other,  as  well  as  tested  for  their  ability  to  predict  the  corresponding  unit’s 
ORI  rating  (the  response  variable).  These  results  are  outlined  in  Chapter  4  of  this  paper. 

In  order  to  create  an  accurate  picture  of  the  state  of  an  organization’s  health,  it  is 
necessary  to  plot  the  KPI  data  for  several  periods  leading  up  to  the  current  period.  The 
reason  for  this  is  to  analyze  trends  to  detennine  where  the  organization  has  come  from, 
where  it  is  today,  and  predict  which  way  it’s  headed  in  the  near  future.  For  the  ACC  case 
study,  SORTS  scores  for  individual  units  should  be  plotted  over  the  previous  six  months 
in  order  to  sufficiently  detennine  trends  in  the  data.  The  current  month’s  data  for  each 
organization  will  highlight  the  shortfalls  between  their  required  readiness  level  and  its 
actual  capability  to  perform  its  mission.  This  shortfall  is  also  known  as  a  capability  gap. 

Once  the  readiness  capability  gaps  have  been  identified  for  each  unit  within  the 
organization,  each  unit  should  be  prioritized  based  on  the  largest  capability  gaps.  This 
will  identify  those  units  that  are  most  in  need  of  continuous  process  improvement.  At  the 
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working  level,  this  simply  means  sorting  the  capability  gaps  in  Excel  from  largest  to 
smallest. 

In  order  to  make  the  previous  work  worthwhile  and  actionable,  the  results  must  be 
communicated  clearly  to  the  decision  maker.  The  key  here  is  to  present  the  maximum 
amount  of  infonnation  possible  while  ensuring  unambiguous  understanding  of  the 
insights  it  presents  (Tufte,  1997).  This  step  entails  presenting  the  plots  for  both  the 
current  period  and  the  previous  periods  in  order  to  portray  the  current  state  of  the 
organization  to  the  decision  maker  and  highlight  the  units  with  the  largest  capability  gaps. 
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IV.  Analysis  &  Results 


Assumptions 

The  data  set  examined  is  comprised  of  Active  Duty  Status  of  Resources  and 
Training  (SORTS)-reporting  units  that  were  part  of  Air  Combat  Command  (ACC) 
between  October  2008  and  October  2010.  This  includes  data  for  units  that  belong  to  the 
8th  Air  Force,  which  were  previously  under  ACC  but  are  now  under  Air  Force  Global 
Strike  Command  (AFGSC).  Data  for  Air  National  Guard  (ANG)  and  Air  Force  Reserve 
Command  (AFRC)  units  were  not  included  in  this  study. 

limitations 

SORTS  scores  are  lagging  indicators,  which  means  that  they  measure  the  output 
of  past  activity.  As  noted  in  Chapter  2  of  this  paper,  these  scores  are  useful  for  tracking 
trends,  but  by  the  time  a  negative  trend  is  noticed,  there  may  already  be  problems. 

Another  limitation  that  surfaced  during  this  study  was  the  fact  that  ACC  reports 
SORTS  scores  for  aviation  units  and  support  units  separately,  on  an  alternating  bi¬ 
monthly  basis  (i.e.,  scores  for  aviation  units  are  reported  during  odd  months,  and  scores 
for  support  units  are  reported  during  even  months).  And  since  SORTS  data  are  not 
archived  on  the  SORTS  website,  only  half  of  the  complete  data  set  was  available  for  this 
study. 

Defense  Readiness  Reporting  System  (DRRS)  and  Air  Expeditionary  Force 
(AEF)  Unit  Type  Code  (UTC)  Status  Reporting  Tool  (ART)  scores  were  introduced 
because  of  their  usefulness  to  assess  readiness.  However,  the  Air  Force  deploys  units, 
entities,  etc.  on  the  UTC  system,  so  the  deployed  units  measured  by  DRRS  and  ART  are 
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not  the  same  as  the  home  units  measured  by  other  systems,  such  as  Operational 
Readiness  Inspections  (ORIs)  and  SORTS.  Since  this  would  be  like  “comparing  apples 
to  oranges”,  the  DRRS  and  ART  data  were  not  included  in  this  study.  In  addition, 
historical  ART  data  is  not  currently  being  maintained,  so  it  was  also  unavailable. 

There  were  also  limitations  to  performing  regression  analysis  in  this  study.  ORI 
scores  are  reported  at  the  wing  level  only;  they  are  not  an  aggregate  of  scores  for  the 
squadrons  that  comprise  the  wing.  Similarly,  SORTS  scores  are  only  assessed  at  the 
squadron  level;  they  are  not  aggregated  up  to  a  wing-level  score.  Therefore,  SORTS  and 
ORI  scores  are  not  directly  comparable,  which  severely  limits  the  ability  to  draw 
conclusions  about  correlation  and/or  predictability  between  the  different  types  of  scores. 
Several  methods  were  used  in  attempt  to  work  around  this  issue,  but  to  no  avail. 

Actual  SORTS  scores  for  individual  units  and  specific  results  about  trends  and 
capability  gaps  are  classified.  These  results  will  be  replaced  with  notional  data  in  order 
to  communicate  the  methodology  while  preserving  the  sensitivity  of  the  data.  All  other 
results  are  unclassified. 

Analysis  &  Results 

Once  all  of  the  SORTS  and  ORI  data  were  collected  and  combined  into  one 
database,  it  was  necessary  to  eliminate  incomplete  and  erroneous  SORTS  observations 
from  the  data  set.  This  included  observations  from  units  that  no  longer  belong  to  ACC, 
observations  that  did  not  have  a  score  in  one  or  more  of  the  P,  T,  S,  or  R  categories,  and 
observations  where  any  of  the  categories  were  reported  as  C-6.  Recall  from  Chapter  3 
that  C-6  is  not  a  rating  and  may  not  be  used  as  an  overall  C-level  because  the  unit  is  not 
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required  to  measure  assets  in  a  specified  area  (AFI  10-201,  2006).  The  remaining  data 
set  was  comprised  of  over  1,300  usable  observations,  as  depicted  in  Table  4  below: 


Table  4.  Observation  Breakdown 


Total 

#  ORIs 

12 

Total  Observations 

1905 

Non-ACC  Observations 

170 

Bad  Observations 

393 

Good  ACC  Observations 

1342 

Once  the  data  set  was  purged  of  “bad”  data,  the  first  step  was  to  use  multivariate 
analysis  to  examine  the  SORTS  and  ORI  scores  for  interdependencies.  A  correlation 
model  was  used  to  identify  any  possible  causal  relationship  between  the  P,  T,  S,  and  R 
variables.  A  high  correlation  between  any  pair  of  these  variables  could  identify  the 
potential  to  impact  one  variable  based  on  a  change  in  another  variable.  For  example,  if  P 
and  T  scores  were  correlated,  this  would  imply  that  improving  a  unit’s  personnel 
availability  may  also  affect  (and  hopefully,  improve)  that  unit’s  training  status.  This 
hypothesis  seems  reasonable  because  people  can  only  be  trained  if  they  are  available. 
Correlation  between  the  other  variables  could  be  used  to  make  similar  interpretations. 

As  it  turned  out,  there  was  no  apparent  correlation  between  any  of  the  SORTS 
variables.  All  correlation  coefficients  were  below  0.25,  indicating  very  low  correlation. 
Table  5  below  shows  the  correlation  matrix  for  the  SORTS  scores. 
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Table  5.  P-T-S-R  Correlation  Matrix 


P 

T 

S 

R 

p 

1 

-0.1751 

-0.0654 

-0.0282 

T 

-0.1751 

1 

0.1195 

0.0623 

S 

-0.0654 

0.1195 

1 

0.2360 

R 

-0.0282 

0.0623 

0.2360 

1 

The  next  effort  was  to  examine  the  ability  of  SORTS  scores  to  predict  ORI 
ratings.  In  addition  to  the  specific  rating,  the  categories  were  broadened  to  assess  the 
model’s  ability  to  predict  whether  the  wing  passed  or  failed  each  inspection. 

Specifically,  a  rating  of  “Outstanding”,  “Excellent”,  or  “Satisfactory”  is  considered  to  be 
passing,  while  a  rating  of  “Marginal”  or  “Unsatisfactory”  is  considered  a  failure. 

However,  since  SORTS  and  ORI  rankings  are  reported  at  different  levels 
(squadron  and  wing,  respectively),  a  workaround  was  used  that  involved  applying  a 
wing’s  ORI  score  onto  all  of  the  squadrons  in  that  wing  during  the  month  of  the  ORI. 

For  example,  if  the  ORI  rating  for  wing  A  (which  is  comprised  of  squadrons  1,  2,  and  3) 
is  “Satisfactory”  in  January  2010,  then  the  rating  for  squadrons  1,  2,  and  3  in  January 
2010  is  “Satisfactory”.  The  purpose  for  doing  this  was  to  have  directly  comparable  data 
(i.e.,  “apples  to  apples”),  as  well  as  to  generate  more  useable  data  points  for  the 
correlation  and  regression  models.  This  enabled  us  to  expand  the  12  ORIs  to  49 
ORESORTS  observations.  The  results  for  this  method  applied  to  the  responses  of  ORI 
rating  and  pass/fail  are  shown  in  Table  6  below: 
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Table  6.  Regression:  Using  SORTS  Scores  to  Predict  ORIs 


ORI  Rating 

ORI  Pass/Fail 

Predictor 

Obs 

R2 

Source  Effect 

Correlation 

Coefficient 

R2 

Source  Effect 

P  Score 

>0.1 

0.1647 

0.0390 

S  Score 

49 

0.1485 

0.0333 

0.2312 

0.2659 

>0.1 

T  Score 

>0.1 

0.0502 

>0.1 

R  Score 

0.0622 

-0.0095 

>0.1 

2 

These  results  show  that  the  R  values  for  both  models  were  very  low,  which 
indicates  that  neither  model  was  capable  of  accurately  predicting  ORI  results.  The  low  p- 
values  indicate  that  the  source  effect  of  the  S  and  R  scores  have  comparatively  more 
impact  on  the  ORI  rating  than  the  P  and  T  scores,  but  the  low  correlation  coefficients 
indicate  that  there  was  very  little  dependence  between  any  of  the  scores  and  the  ORI 
rating.  The  P  score  seems  to  have  more  of  an  effect  on  whether  the  unit  passes  the  ORI, 
but  the  low  R”  value  invalidates  any  assumed  predictive  power.  (Note:  There  are  no 
correlation  coefficients  for  the  Pass/Fail  model  because  the  response  variable  was  based 
on  a  character  nominal  scale.) 

We  also  used  various  averages  for  the  squadron  SORTS  scores  in  an  attempt  to 
reduce  variability  within  the  individual  scores  and  capture  values  that  incorporate  the 
previous  months  leading  up  to  the  ORI.  These  variations  include:  taking  the  average 
across  the  P,  T,  S,  R  scores  for  each  inspected  wing,  the  average  of  each  of  the  P,  T,  S,  R 
scores  for  all  squadrons  within  each  inspected  wing  during  the  month  of  the  ORI,  and  the 
average  for  all  squadrons  within  each  inspected  wing  over  each  month  from  one  to  six 
months  prior  to  the  inspection.  All  of  these  attempts  yielded  results  similar  to  the  first 
iteration  presented  above  (see  Table  7  below). 
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Table  7.  Regression:  Using  SORTS  Averages  to  Predict  ORIs 


ORI  Rating 

ORI  Pass/Fail 

Predictor 

Obs 

R2 

Source  Effect 

Correlation 

Coefficient 

R2 

Source  Effect 

Sq  Scores  -  Avg  Across 

49 

<0.1 

0.0371 

0.2337 

<0.1 

>0.1 

Wg  Avg  -  same  month 

48 

<0.1 

>0.1 

0.0282 

<0.1 

>0.1 

1-mo  Avg 

44 

<0.1 

>0.1 

<  .0001 

<0.1 

>0.1 

2- mo  Avg 

44 

<0.1 

>0.1 

0.0631 

<0.1 

>0.1 

3- mo  Avg 

44 

<0.1 

>0.1 

0.1194 

<0.1 

>0.1 

4- mo  Avg 

40 

<0.1 

>0.1 

0.1252 

<0.1 

>0.1 

5- mo  Avg 

40 

<0.1 

>0.1 

0.1417 

<0.1 

>0.1 

6- mo  Avg 

40 

<0.1 

>0.1 

0.1435 

<0.1 

>0.1 

Unfortunately,  the  inherent  problem  with  this  approach  was  that  the  four  predictor 
variables,  which  had  considerable  variance,  were  used  to  predict  a  response  variable  with 
no  variance.  This  severely  limited  the  predictive  capability  of  the  model.  (A 
recommendation  to  report  ORI  rankings  and  SORTS  scores  at  both  the  squadron  and 
wing  levels  will  be  recorded  in  Chapter  5.) 

Trends  and  Capability  Gaps 

After  examining  the  dependencies  within  the  data,  the  next  step  was  to  plot  the 
data  in  order  to  analyze  trends  and  identify  capability  gaps.  In  general,  a  capability  gap 
can  be  measured  by  comparing  the  actual  scores  against  a  target  value  which  is  based  on 
the  decision  maker’s  goals  for  the  organization.  For  this  study,  the  target  SORTS  score 
was  2  because  ACC’s  goal  is  for  units  to  be  C-2  or  better.  What  ACC  would  like  to  do  is 
minimize  the  maximum  P,  T,  S,  R  score  for  any  unit,  which  is  referred  to  as  a  minimax 
problem.  (Winston,  2004)  However,  by  simply  reporting  that  minimax,  much 
information  is  lost.  Since  the  overall  SORTS  C-level  is  the  maximum  of  the  P,  T,  S,  R 
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scores,  a  unit’s  overall  score  is  not  necessarily  representative  of  its  actual  health  because 
not  all  sub-categories  are  reflected  in  the  score.  (A  recommendation  to  change  the 
method  for  determining  overall  SORTS  C-levels  is  recorded  in  Chapter  5.)  This  issue 
was  alleviated  for  this  study  by  taking  the  average  across  the  scores.  This  average  doesn't 
have  any  mathematical  meaning  because  it  was  taken  across  categories  that  measure 
different  things,  but  it  can  be  used  as  a  metric  that  distinguishes  between  units  that  have 
the  same  overall  (max)  score. 

Table  8  below  illustrates  this  method.  Suppose  Squadron  A  has  P,  T,  S,  R  scores 
of  4-1- 1-1  and  Squadron  B  has  scores  of  4-4-4-4.  Both  have  the  same  overall  SORTS 
score  (C-4),  but  the  reader  can  see  that  A  is  actually  doing  better  than  B.  Taking  the 
average  of  A’s  and  B’s  scores  (1.75  and  4.0,  respectively)  gives  a  much  better  feel  for 
how  well  each  unit  is  actually  able  to  perform  its  mission.  In  addition,  the  average  can  be 
used  to  determine  each  unit’s  capability  gap,  which  shows  how  close  the  unit’s  average 
score  is  from  being  C-2.  In  this  example,  Squadron  A’s  gap  is  -0.25  and  B’s  gap  is 
2.0.  (The  negative  gap  indicates  that  a  unit  is  actually  performing  better  than  the  target.) 


Table  8.  Determining  a  Unit's  Overall  Health 


Sq 

P-T-S-R 

C-level 

(Max) 

Avg 

Target 

Capability  Gap 
(Avg  minus  Target) 

A 

4- 1-1-1 

4 

1.75 

2 

-0.25 

B 

4-4-4-4 

4 

4 

2 

2 

To  apply  continuous  process  improvement  effectively,  the  decision  maker  should 
apply  his  resources  using  a  systematic  approach.  He  should  prioritize  the  organizational 
units  for  improvement  efforts  according  to  the  size  of  their  capability  gaps.  A  key  aspect 
that  enables  this  prioritization  to  work  is  the  fact  that  the  average  scores  for  the  lowest- 
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level  units  can  easily  be  aggregated  to  give  the  decision  maker  insight  about  his 
organization  at  all  levels.  In  other  words,  the  average  score  for  any  level  in  the  hierarchy 
is  the  average  of  all  lowest-level  units  under  that  level.  For  example,  for  ACC,  the  Major 
Command  (MAJCOM),  Numbered  Air  Force  (NAF),  and  Wing  averages  are  simply  the 
averages  of  all  squadrons  contained  within  each  of  those  organizations.  Figure  9  shows 
this  relationship  hierarchically.  In  this  case,  the  NAF  1  Average  is  the  average  of  Wings 
1-4,  the  NAF  2  Average  is  the  average  of  Wings  5-7,  and  the  NAF  3  Average  is  the 
average  of  Wings  8-11.  Similarly,  the  MAJCOM  Average  is  the  average  of  Wings  1-11. 
All  wing  averages  are  the  average  of  all  squadrons  contained  in  them  (not  depicted). 


Figure  9.  Aggregation  of  Squadron  SORTS  Scores 


Since  ACC’s  actual  SORTS  scores  and  the  specific  results  of  this  study  are 
classified,  the  following  scenario  will  be  used  to  communicate  how  this  method  of 
prioritization  was  used. 

Suppose  that  on  January  10,  2010,  a  NAF  commander  wanted  to  identify  and 
improve  the  worst  performing  squadron  in  his  organization.  The  NAF  structure  is 
presented  below  in  Figure  10. 
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Figure  10.  Notional  NAF  Structure 


The  first  thing  his  team  did  was  to  plot  the  average  squadron  SORTS  scores  based 
on  available  data  for  Wings  1  and  2  from  February  2009  through  December  2009  (see 
Figures  11  and  12  below). 


Wing  1  (Feb-Dec  09) 


•  Target 
-Wing  Avg 


Figure  11.  Plot  of  Notional  SORTS  Scores  for  Wing  1 
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Figure  12.  Plot  of  Notional  SORTS  Scores  for  Wing  2 


The  commander’s  team  ascertained  that  Wing  1,  with  the  exception  of  a  late  spike 
in  November,  was  operating  at  a  fairly  steady  level  in  line  with  the  perfonnance  target. 
Wing  2,  however,  was  getting  considerably  worse  after  performing  nominally  through 
September.  Therefore,  the  team  decided  to  dig  deeper  into  Wing  2  by  examining  the 
performance  of  its  sub-organizations,  Squadrons  3  and  4.  Figures  13  and  14  below  are 
plots  of  the  notional  SORTS  data  for  Squadrons  3  and  4,  respectively. 


Squadron  3  (Feb-Dec  09) 


Figure  13.  Plot  of  Notional  SORTS  Scores  for  Squadron  3 


Squadron  4  (Feb-Dec  09) 
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Figure  14.  Plot  of  Notional  SORTS  Scores  for  Squadron  4 


The  commander’s  team  realized  they  had  more  visibility  into  the  overall  health  of  the 
squadrons  by  looking  at  the  average  SORTS  scores  in  addition  to  the  C-level.  For 
Squadron  3,  the  C-level  remained  constant  at  4  throughout  the  entire  time  period,  but  the 
average  revealed  the  fluctuation  throughout  the  time  period.  For  Squadron  4,  the  trend 
for  the  C-level  actually  mirrored  that  of  the  squadron  average  (both  started  getting  worse 
between  June  and  August),  but  it  still  did  not  adequately  capture  Squadron  4’s  overall 
health.  In  this  case,  the  squadron  average  was  always  below  the  target  value  C-2,  which 
revealed  that  the  overall  health  of  Squadron  3  was  worse  than  that  of  Squadron  4. 
Therefore,  the  commander’s  team  recommended  doing  a  CPI  event  on  Squadron  3  before 
working  on  Squadron  4.  The  commander,  who  saw  the  logic  in  the  approach,  approved 
the  team’s  recommendation. 
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V.  Recommendations  &  Conclusion 


Summary 

In  chapters  1  and  2  of  this  paper,  the  reader  was  introduced  to  the  concepts  of 
continuous  process  improvement,  metric  development,  and  regression  analysis.  In 
chapter  3,  these  concepts  were  fused  to  produce  a  methodology  for  improving  the  bottom 
line  of  an  organization  by  improving  its  weakest  performance  areas.  The  usefulness  of 
this  methodology  was  demonstrated  through  practical  application  to  ACC  in  chapter  4. 

Recommendations  for  Future  Study 

The  key  perfonnance  indicators  analyzed  in  this  paper  are  not  all-inclusive.  For 
the  ACC  case  study,  which  can  be  applied  to  any  MAJCOM  in  the  Air  Force,  scores  from 
other  readiness  reporting  systems  can  serve  as  potential  readiness  indicators.  The  idea  of 
using  DRRS  and  ART  scores  was  introduced  in  Chapter  3.  Another  set  of  metrics  worthy 
of  examination  are  Unit  Compliance  Inspection  (UCI)  scores.  UCIs  are  conducted  to 
assess  mission  areas  that  are  critical  to  the  health  and  performance  of  an  organization, 
and  a  unit’s  failure  to  comply  with  these  areas  could  result  in  legal  liabilities,  penalties,  or 
mission  impact.  Therefore,  it  seems  logical  to  conclude  that  a  non-compliant  unit  is  not 
ready  to  perfonn  its  mission,  and  hence  UCI  scores  could  be  used  to  assess  readiness. 

Since  SORTS  scores  are  fairly  subjective,  it  may  be  worthwhile  to  compare 
SORTS  P  ratings  for  individual  squadrons  versus  the  manning  levels  for  the  same 
squadrons  during  those  months.  This  would  provide  insight  into  the  fidelity  of  the 
SORTS  scale  by  checking  the  consistency  with  which  the  ratings  are  currently  being 
applied. 
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Finally,  it  may  be  beneficial  to  perform  the  regression  and  trends  analysis  on 
aviation  and  support  units  separately.  Since  a  fighter  squadron  has  a  very  different 
mission  from  a  medical  squadron,  for  example,  each  unit  would  likely  use  different 
definitions  for  mission  impact  and  would  assess  their  readiness  accordingly.  This  would 
likely  result  in  a  disparity  between  their  respective  scores,  which  would  essentially 
appear  as  though  they  are  using  two  completely  different  scales.  Studying  the  two  types 
of  units  separately  may  potentially  improve  the  fidelity  of  both  models. 

Other  Recommendations 

The  following  recommendations  are  intended  to  overcome  some  of  the  limitations 
that  were  mentioned  in  Chapter  4  of  this  paper.  First,  the  fact  that  ACC  reports  SORTS 
scores  for  aviation  and  support  units  on  an  alternating  bi-monthly  basis,  coupled  with  the 
issue  that  SORTS  data  are  not  archived  on  the  SORTS  website,  imposed  a  limitation  on 
the  amount  of  data  that  was  available  for  this  study.  Therefore,  it  is  recommended  that  all 
SORTS  and  ART  data  be  archived  on  its  website  for  easy  retrieval.  In  the  meantime,  it 
would  be  helpful  if  ACC  would  report  all  SORTS  scores  every  month  until  the  website  is 
updated.  In  addition,  functionality  should  be  added  that  allows  a  user  to  produce 
customized  reports. 

There  were  also  limitations  to  performing  regression  analysis  in  this  study.  Since 
ORI  scores  are  only  reported  at  the  wing  level  and  SORTS  scores  are  only  assessed  at  the 
squadron  level,  these  scores  were  not  directly  comparable.  One  possible  option  for 
overcoming  this  limitation  would  be  to  aggregate  squadron  SORTS  data  into  an  overall 
wing  score.  This  would  allow  both  SORTS  and  ORI  scores  to  be  compared  at  the  wing 
level.  Another  option  would  be  to  assess  ORIs  at  the  squadron  level,  which  would  also 
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allow  for  direct  comparison  of  both  scores.  Having  this  data  readily  accessible  on  the 
respective  websites  would  also  be  beneficial. 

Another  limitation  encountered  during  this  research  was  the  current  method  for 
detennining  overall  SORTS  scores.  Currently,  a  unit’s  overall  SORTS  C-level  is  the 
maximum  of  its  P,  T,  S,  R  scores.  As  we  saw  in  Chapter  4,  however,  the  maximum  score 
is  not  necessarily  representative  of  a  unit’s  overall  health.  Therefore,  it  is  recommended 
that  the  method  for  determining  overall  SORTS  C-levels  should  become  something 
besides  the  maximum  of  the  P,  T,  S,  R  scores.  The  workaround  used  in  this  study 
involved  taking  the  average  across  the  four  scores,  but  this  is  certainly  not  the  only 
option. 

The  AFS02 1  tool  kit  is  powerful,  flexible,  and  extremely  useful.  It  is  critical  that 
it  should  be  used  today  in  a  way  that  will  help  the  Air  Force  do  its  mission  better 
tomorrow. 
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Appendix  A:  Abbreviations 


ACC  -  Air  Combat  Command 

ACC/A3  -  Air  Combat  Command  Directorate  of  Operations 

ACC/A9  -  Air  Combat  Command  Directorate  of  Analyses,  Lessons  Learned  &  AFS02 1 

AEF  -  Air  Expeditionary  Force 

AFI  -  Air  Force  Instruction 

AFPD  -  Air  Force  Program  Directive 

AFS021  -  Air  Force  Smart  Operations  for  the  21st  Century 

ART  -  AEF  (Air  Expeditionary  Force)  Unit  Type  Code  (UTC)  Status  Reporting  Tool 

BPR  -  Business  Process  Reengineering 

C  -  Category 

CAF  -  Combat  Air  Force 

CJCS  -  Chainnan  of  the  Joint  Chiefs  of  Staff 

COCOM  -  Combatant  Commander 

COMACC  -  Commander  of  Air  Combat  Command 

CPI  -  Continuous  Process  Improvement 

CSAF  -  Chief  of  Staff  United  States  Air  Force 

DMAIC  -  Define,  Measure,  Analyze,  Improve,  Control 

DoD  -  Department  of  Defense 

DRRS  -  Defense  Readiness  Reporting  System 

IG  -  Inspector  General 

KPI  -  Key  Performance  Indicator 

LSS  -  Lean  Six  Sigma 

MOE  -  Measure  of  Effectiveness 

MOP  -  Measure  of  Performance 

NMS  -  National  Military  Strategy 

ORI  -  Operational  Readiness  Inspection 

OSD  -  Office  of  the  Secretary  of  Defense 

P  -  Personnel 

R  -  Equipment  Condition 

S  -  Supplies/Equipment 

SA&D  -  Strategic  Alignment  and  Deployment 

SIPRNet  -  Secret  Internet  Protocol  Router  Network 

SORTS  -  Status  of  Resources  and  Training  Reporting  System 

T  -  Training 

TOC  -  Theory  of  Constraints 
TQM  -  Total  Quality  Management 
UMD  -  Unit  Manning  Document 
UTC  -  Unit  Type  Code 
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Appendix  B:  Blue  Dart 


AFS021 :  How  do  we  make  it  work? 

Capt  Mark  Degenhardt 
Dept,  of  Operational  Sciences 
Air  Force  Institute  of  Technology 

Air  Force  Smart  Operations  for  the  21st  Century  (AFS021)  is  not  currently  being 
used  as  effectively  as  it  could  be  across  the  Air  Force.  Thanks  to  the  Air  Force  website, 
Air  Force  Times,  and  other  media  sources,  we  hear  about  various  success  stories  where 
processes  are  streamlined,  hundreds  of  man-hours  are  cut,  and  thousands  of  American  tax 
dollars  are  saved.  These  are  obviously  very  good  things,  but  we  need  to  ask  ourselves 
what  effect  these  scattered  (and  somewhat  random)  improvements  have  on  the  core 
mission?  For  example,  did  streamlining  an  F-16  tire  manufacturing  process  to  double  the 
output  capacity  from  four  to  eight  tires  per  day  really  benefit  the  Air  Force?  Did  it  enable 
us  to  put  more  planes  on  the  runway  and  into  the  fight?  How  was  this  particular  process 
chosen?  What  was  the  reason  for  doing  the  event?  Was  there  a  reason?  Or  was  the  unit 
directed  to  “do  an  AFS021  event  on  something”  just  to  meet  a  quota? 

All  of  these  queries  lead  to  one  underlying  question:  how  do  we  use  AFS02 1  to 
go  from  creating  OPR  fodder  to  truly  making  a  difference  for  the  Air  Force?  This  can  be 
compared  to  a  medical  patient  with  a  minor  but  lingering  illness  asking  a  doctor  how  to 
get  better.  The  doctor  would  probably  tell  the  patient  that  they  need  to  stop  trying  to  treat 
the  symptoms  and  start  treating  the  root  causes  of  their  problem.  For  the  Air  Force,  this 
means  that,  instead  of  trying  to  simply  save  money  here  or  cut  man-hours  there,  AFS02 1 
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tools  should  be  aimed  at  solving  the  bigger  issue  of  improving  the  Air  Force’s  bottom 
line.  For  a  business,  the  bottom  line  is  profit.  Therefore,  to  improve  their  bottom  line  is 
to  increase  profit.  However,  since  the  Air  Force  is  a  non-profit  government  entity,  it  can 
be  argued  that  our  bottom  line  is  mission  perfonnance.  In  other  words,  we  need  to  use 
these  tools  to  “fly,  fight  and  win”  better.  As  it  turns  out,  the  Air  Force  Institute  of 
Technology  (AFIT)  has  done  some  research  that  addresses  this  very  issue. 

Through  its  research,  AFIT  developed  a  methodology  to  identify  specific  target 
areas  where  continuous  process  improvement,  i.e.,  AFS021,  should  be  applied  to 
improve  the  bottom  line  of  an  organization.  The  first  step  in  this  process  is  to  engage  the 
decision  maker  to  solicit  the  key  perfonnance  indicators  (KPIs)  that  best  reflect  the 
organization’s  mission.  The  goal  for  this  step  is  to  make  a  list  of  the  answers  to  the  “why 
are  we  here”  question.  The  second  step  is  to  use  and/or  develop  appropriate  metrics 
based  on  those  KPIs  to  measure  how  well  the  organization  is  doing  its  mission  today. 

The  third  step  is  to  capture  the  trends  of  those  KPIs  over  time  to  see  if  the  organization  is 
getting  better  or  worse.  The  fourth  and  final  step  is  to  identify  the  largest  performance 
capability  gaps  in  order  to  determine  where  AFS02 1  resources  should  be  applied  to  get 
“the  most  bang  for  the  buck”.  The  result  of  this  process  should  give  the  decision  maker  a 
clear  snapshot  of  his/her  organization’s  current  ability  to  perform  its  mission.  This,  in 
turn,  gives  him/her  the  ability  to  improve  the  bottom  line  by  improving  the  weakest  areas. 

The  AFS02 1  tool  kit  is  powerful,  flexible,  and  extremely  useful.  We  should  be 
using  it  today  in  a  way  that  will  make  the  Air  Force  do  its  mission  better  tomorrow. 
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Metric  Development  for 
Continuous  Process  Improvement 


Appendix  C:  Storyboard 
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